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Abstract 

In this paper we offer a formal, rigorous proof of the correctness of Awerbuch's algorithm for 
network synchronization [1]. We specify both the algorithm and the correctness condition using 
the I/O automaton model. Our proof of correctness follows Awerbuch's intuitive arguments 
closely by exploiting the model's natural support for techniques of stepwise refinement and 
modularity. We demonstrate that the model is a powerful tool for reasoning about distributed 
algorithms. 
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1 Introduction 

We prove the correctnesss of a distributed protocol for network synchronization. Most networks 
do not offer reliable bounds on the time a message takes to arrive, so it is important to find 
algorithms that work correctly in an asynchronous system. It is, however, much easier to design 
algorithms if the network is synchronous. Awerbuch [1] proposed the use of a synchronizer that 
would enable one to convert any synchronous graph algorithm into an algorithm that performs 
correctly in an asynchronous (but failure-free) network. Such synchronizers have been used to give 
efficient asynchronous algorithms ([2]). We give a formal proof of correctness of this algorithm. 

In [1], a synchronizer (called 7 in that paper) is constructed for a network whose topology is 
any fixed, connected graph. The algorithm is described in terms of sub-algorithms that run on 
trees of a suitable spanning forest subgraph, and a distributed technique is given for finding a such 
a subgraph for which the resulting algorithm has low time and message complexity. 

The synchronizing algorithm is derived as a composition of a simple synchronizer (called /3) 
executing within each "cluster" (the set of nodes in each tree of the spanning forest subgraph), 
and another simple synchronizer (called a) that synchronizes between the clusters. While this 
description helps to explain the detailed algorithm, no formal proof of correctness is offered in [1]. 
We provide a formal account of an algorithm closely based on Awerbuch's description, and rigor- 
ously prove results about its correctness. Our exposition follows Awerbuch's informal arguments 
by building on claims that express formally the correctness of algorithms a and /3 that work at the 
level of the clusters and in between them (the spanning forest subgraph that defines these clusters 
is assumed to be given). The proof is modular and hierarchical, and uses the I/O automaton for- 
malism to define the problem and to describe and prove the correctness of the algorithm offered as 
solution. The reader is referred to [3] for a treatment of I/O automata. 

We define synchronizing behavior by specifying an I/O automaton that uses global information 
about the system to coordinate the behaviors of all the client automata. Our goal is to arrive at 
algorithms (automata) running on each of the nodes of the networks that, acting together, emulate 
the behavior of the global synchronizer. More precisely, we will produce a collection of automata 
which is such that from the point of view of each client, executions implemented by the collection 
are indistinguishable from those resulting from the presence of the global synchronizer. 

First, we provide a specification for a global network synchronizer by giving a single I/O au- 
tomaton that synchronizes across the network by exchanging messages directly with each node in 
the network. We then repeatedly refine this specification according to Awerbuch's construction. At 
each step, we will specify a collection of automata that will "implement" the synchronizing behavior 
of the global synchronizer in progressively "more distributed" settings. We will state and prove 
the corresponding correctness claims that justify each refinement, and finally arrive at a collection 
of automata that will run on individual nodes of the network - a collection that, we will prove, 
continues to synchronize the network. 

After a brief description of the setting, notations and conventions in Section 1.1, we proceed in 
Section 2 to give a general specification for client automata, which are node automata that execute 
steps of a given synchronous graph algorithm that we wish to execute on the network. We then 
define the previously described global synchronizer globjynch. Having thus laid out the problem 
and goal, in Section 2.2 we detail a collection of automata that "implements" glob^synch. One of 
the components of the collection is a "local" synchronizer (called locsynch in this report), which 



ensures that adjacent nodes are always within one synchronous-algorithm timestep of each other 
at all times. 

We prove in Section 2.3 that, as a consequence, no client can distinguish globsynch from the 
collection, and hence that the collection is, in a weak sense, an implementation of globsynch. 

A stronger (and conventional) notion automaton implementation involves behavior inclusion: 
We say that the composition C of a collection of automata implements an automaton A, if every 
behavior of C is a behavior of A. 

It is this definition that will apply in subsequent sections when we consider the task of imple- 
menting locsynch in a distributed setting. 

In Section 3.1, we describe how locsynch is implemented at the level of each of Awerbuch's clus- 
ters, and in between clusters. We prove that every external behavior of the composition of automata 
described is a behavior of locsynch. In later sections, we provide distributed implementations and 
the necessary proofs of implementations of the intracluster and intercluster synchronizers that real- 
ize locsynch. We dwell briefly on some highlights of the proof presented and end with a summary 
in Section 4. 

1.1 Preliminaries 

In this section, we describe the setting of the problem, and introduce some conventions and notations 
to be used throughout this report. 

We are working with an asynchronous, failure-free network, whose topology is described by a 
fixed, connected, communication graph G = (V, E), where the set of nodes V represents processors 
of the network and the set of links E represents the communication channels between them. Mes- 
sages can be exchanged between pairs of neighboring nodes, and arrive in the order in which they 
are sent with finite but unbounded delays. 

We assume a basic data type message. Ordered pairs of the form (m, q) , where m is a message 
and q £ V is a vertex, are said to be of type tagged message, and we refer to (m, q) as the tagged 
message tagged by q. We use e to denote the null message. 

We use "*" to denote a parameter we allow to take any permitted value of the right type 
(Restrictions on the the type and the values will be clear from the context). 

With explicitly noted exceptions, Greek letters 7r, p and a are used to denote actions of I/O 
automata, and /3, fl', 7 and 8 to represent sequences of those actions. 

We call neigh(p) the set {q £ V\(p, q) £ E}. Wherever used, i and j are always nonnegative 
integers. 

To understand the functioning of client automata, we will need the idea of preservation of 
certain properties by clients. We provide a formal definition here, deferring motivations for use to 
the next section, where clients are discussed. 

Definition 1.1 Let M be an I/O automaton and V (the set of properties to be preserved) be a 
non-empty, prefix-closed set of sequences of actions from a set # satisfying $ H int(M) = <f>. M is 
said to preserve V i//37r|$ £ V whenever /3|$ £ V , n £ out(M), and (3tt\M £ finbehs(M). 

Variables s and t denote states of automata. The following definition describes our notation for 
talking about states of compositions of automata. 



Definition 1.2 Let A = J\i Ai be a composition of the collection of automata {Ai}. Vs 6 states(A), 
s[Ai] is the state of component automaton Ai when A is in state s. 

Throughout this report, unless otherwise specified, all arrays of automata state variables store 
values of type boolean, and all I/O automata implementations are such that all boolean variables 
are initially false, and once set to true by an action, remain true forever thereafter during that 
execution. With one exception (the client automata, Section 2.1.1), all I/O automata implemented 
in this report have no internal actions, are completely deterministic and are such that once any of 
their output actions is enabled, it is disabled only if it is performed. 

In the next section we lay out the basic problem of network synchronization using the I/O 
automaton formalism. 

2 The Network Synchronization Problem 

In this subsection, we will formalize the network synchronization problem. 

2.1 A specification of a globally synchronous system 

With each node p £ V, we will associate an automaton client(p) that, we will assume, runs the 
rounds of the synchronous algorithm we wish to execute on the network. 

In this section we will give an informal overview of synchronizing behavior and the workings 
of client automata and globjsynch, and then translate that description into the language of I/O 
automata. 

The client automata are assumed to follow a protocol whereby, at the end of each round, they 
each output a tagged message set that contains all outgoing messages, and then wait for an input 
action delivering incoming messages before commencing computations for the subsequent round. 

The automaton globjsynch ensures synchronous execution of the underlying algorithm by wait- 
ing until all the clients have output their outgoing tagged message sets before delivering incoming 
messages to any client. 

We now model the above in the framework of the I/O automaton formalism. We begin with 
the specification of the client automata, and follow through with that of globjsynch. 

2.1.1 Client automata 

For concreteness, we assume that each client automaton's external actions consist of an input action 
and an output action. The output carries out round-specific messages to neighboring nodes, and 
the input action conveys that rounds' incoming messages from neighboring nodes to the client. 
Furthermore, we attribute a liveness property to the client automata by requiring them to always 
respond to incoming messages, whenever they are given the chance. This restriction allows us to 
work in the same network-failure free model that is the setting for Awerbuch's algorithm. 

We make the assumption that clients are I/O automata and give below the specification of 
a client automaton at an arbitrary node p £ V (client(p)) that captures the above requirements 
formally. 



Signature 

The client automaton at each node p has the action signature given below. 

Input: 

client -input(p, M, i), M a set of tagged messages 
Output: 

client -Output(p, M,i), M a set of tagged messages 
Internal: 

arbitrary 

The notation for describing action signatures has a straightforward interpretation, and is used 
consistently throughout the remainder of this report. Consider, for example, the client's output 
action. Each client has an output action that, in some unspecified manner, contains three pieces of 
data in it to be conveyed to the automaton sharing this action with the client: A node identifier 
p that, in this case, identifies the client that performs the action, a tagged message set M of the 
indicated type that contains outgoing messages, and i, a positive integer (by convention), that is 
typically interpreted to be the round number of the synchronous algorithm during which the action 
occurred. 

A similar interpretation applies to the client's input action. Internal actions of the client, which 
will include computations for the underlying synchronous algorithm, are left unspecified. 

We continue with the description of the restrictions on the clients. 

For each p E V, associate a set of p-well-formed sequences defined as follows: 

Definition 2.1 A sequence (finite or infinite) is said to be p-well-formed if it consists of alternat- 
ing occurrences of client jOutput(p, *, *) and client -input(p, *, *), starting with client -output (p, *, 1) 
followed by client-input(p, *, 1), such that with each occurrence of client -output (p, *, i), i advances 
by 1, and with each occurrence of client -input(p, *,j), j advances by 1. 

Let W p be the set of p-well-formed sequences. We require the client at node p to preserve W p and 
be such that no fair behavior of that automaton ends in client Jnput(p, *, *). 

By stipulating that clients preserve p-well-formedness, we ensure that clients do not send a 
message "out of turn" - the client is constrained to wait for incoming messages for a round before 
it can send out the outgoing messages for the next round. It is this requirement that supports our 
interpretation of the client as an automaton executing steps of a synchronous algorithm. Clients' 
input actions thus aid in the information flow of the underlying synchronous algorithm, and serves 
as a "clock pulse", telling the client to commence computations for a fresh round. 

The constraint on the last action of a client's behavior guarantees that the client will not simply 
"crash" in the middle of a protocol - an requirement consistent with our assumption that the system 
is failure-free. 

We now proceed to describe the automaton glob^synch that will model synchronous execution 
on the network. 

2.1.2 The global synchronizer 

The global synchronizer (depicted in figure 1) will work as follows. 

After a client automaton finishes the computations that are to be carried out at that node for 
one round of the synchronous algorithm, it sends a client -output message containing all messages 
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Figure 1: The Global Synchronizer and Clients 

to be communicated to its neighbors in the graph to the synchronizing automaton. The automaton 
globsynch waits until it has received similar messages from all the clients, repackages messages by 
destination node, sends to each client one clientAnput message containing all messages from its 
neighbors for that round, and waits for responses in the form of client-output messages of the next 
round. 

Since each of the clients preserve the appropriate well-formedness property, the global synchro- 
nizer's functioning thus ensures synchronous execution of the underlying synchronous algorithm. 

The following implementation of globjsynch makes it work as described. 

Signature 

The automaton shares each input and output action with the clients in the network, and we proceed 
to describe its functioning formally. 

Input: 

client joutput(p, M, i),p £ V, M a set of tagged messages 
Output: 

client -input(p, M,i),p £ V, M a set of tagged messages 
Internal: 



State 

array tray(p, i), p £ V, of sets of tagged messages, initially all empty 
array client-Output-recd(p,i), p £ V 
array clientJ,nput.sent(p,i), p EV 



Transitions 



client -Output(p, M, i) 
Effect: 

s .client joutputjrecd(p, i) = true 

Vg s.tray(q, i) = s' .tray(q, i) U {(m,p)|(m, q) £ M} 

client jinput(p, M, i) 
Precondition: 

Vg £ V, s 1 .client .output _recd(g, i) — true 

M = s' .tray(p, i) 

s' .client-inputsent(p,i) = false 
Effect: 

s .client Jnputsent(p, i) = true 



Partitions 

All client J,nput{p, *, *) actions are in one class, for each p E V. 



Our interpretation of the network synchronization problem is that it is the task of implementing 
the glohjsynch automaton in a distributed setting in a fair manner. That is, provide implementa- 
tions which simulate fair behaviors of glohjsynch. 

In the following two sections we will provide a provably correct fair "implementation" of the 
composition automaton glohjsynch and all the client automaton. 

The automaton glohjsynch works by holding back clients from executing successive steps of 
the algorithm until all clients have completed all message exchanges for previous steps; a "locally 
synchronous" implementation of glohjsynch we are now going to consider will instead have each 
client wait only until all its neighbors have finished their work for the earlier rounds. 

Our motivations for looking at such an implementation are twofold. Firstly, it should be clear 
that waiting until the latter condition is met is substantially easier to realize in a network. That 
aside, the fact that Awerbuch's synchronizer meets this weaker requirement and not that of the 
global synchronizer - this is intuitive - renders this implementation a natural choice owing to its 
potential to be refined into Awerbuch's synchronizer. 

The flip side of this decision, though, is that we have to deal with a collection of automata 
which does not quite "implement" glohjsynch in the usual sense (involving behavioral inclusion). 
What we will establish in the upcoming sections is that the collection we are going to provide 
(we will have called it LF by then) simulates the existence of glohjsynch from the viewpoint of 
each client executing the synchronous algorithm, though each execution may itself not be globally 
synchronous. To be precise, we will show that for every behavior of LF interacting with client 
automata, there exists a behavior of the composition glohjsynch, also interacting with clients, such 
that both behaviors project identically on each client - thus maintaining the semblance of global 
synchrony through local synchronization: 

Theorem Let /3 be any behavior of LF • U p ^vClient(p). Then there exists 
7 G behs{globjsynch ■ R p& vClient(p)) such that for all p £ V, j3\Client{p) = j\Client(p). 



We shall forthwith proceed to the locally synchronous fair implementation of glob^synch. 

2.2 An implementation of the global synchronizer 

The automaton globsynch is modeled as a collection of several automata: One front-end on each 
node that shares actions with the client automaton on that node, two link automata L(p — *■ q) and 
L(q — *■ p) between the front ends of every two clients that happen to be on neighboring nodes p 
and q, and the local synchronizer that we shall call locsynch, that shares messages with all the 
front ends. 

The front ends interface with the client automaton and receive tagged message sets from their 
clients that contain messages to be communicated to client automata on neighboring nodes. Each 
front-end sorts and distributes its own client's outgoing messages (to other front-ends, via the 
bridging link automata), acknowledges receipt of incoming tagged messages thus distributed and 
collates the incoming messages into a tagged message set. The tagged message set is then forwarded 
to the client if the front-end has been granted permission to do so by locsynch. 

When the receipt of all tagged messages sent out has been acknowledged, each front-end noti- 
fies locsynch of this condition (this is the "safety" condition in Awerbuch's description). When 
locsynch receives this signal from a front-end and all its neighbors, it authorizes that front-end 
to forward the tagged message set of incoming messages to its client, that is, it tells the front-end 
to generate the clock pulse that starts off the next round. Figure 2 depicts the various automata 
described and the actions they share among themselves. 

We now give the formal specifications of front-end and link automata, and the local synchronizer, 
in that order, in each case also partitioning locally controlled actions into fairness classes. 
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Figure 2: Automata implementing the global synchronizer 
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2.2.1 Front end automata 

The front-end automaton (front-end(p)) at any node p E V has the following formal specification. 
We remind the reader of our convention that, unless noted otherwise, all arrays of automata state 
variables store boolean values, and that all these values are initialized to be false. 

Input: 

client -joutput(p , M , i) , M a set of tagged messages 
send lnp (q,p, p,i),q £ neigh(p), p, a set of messages 
ack'" p (q,p, i), q £ neigh(p) 

ff°(p,») 
Output: 

client J,nput(p, M, i), M a set of tagged messages 

send"" (p,q, p,i),q £ neigh(p), n a set of messages 

ack out (p, q, i), q £ neigh(p) 

ok(p,i) 
Internal: 



State 

array client -Outputjrecd(i) 

array client -input sent(i) 

array pkt_from(q,i), q 6 neigh(p) 

array pkt-for(q, i), q £ neigh(p) 

array pktsent(q, i), q £ neigh(p) 

array ack_recd(q, i), q £ neigh(p) 

array acksent(q, i), q £ neigh(p) 

array gojrecd(i) 

array oksent(i) 

array outbox(p, i) of sets of messages, initially all empty 

array inbox(i), of sets of tagged messages , initially all empty 
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Transitions 



client -Output(p, M, i) 
Effect: 

s.clientjoutput-recd(i) = true 
Vg such that (m, q) £ M, for some m 
s.outbox(q,i) = {m\(m,q) £ M} 
s.pkt-for(q, i) = true 

8end°"'(p, g,fi,i) 
Precondition: 

a .pfct_se7it(g, i) = false 

s .pkt-for(q,i) = true 

fi = s .outbox(q, i) 
Effect: 

s .pktsent(q, i) = true 

ack ir ">(q,p,i) 
Effect: 

s.acfc_rec<i(g, i) = true 

send'" p (q,p, fi,i) 
Effect: 

s.m6oa:(i) = {(m,g)|m £ /*} U «'.m6ox(i) 
s.pkt-from(q,i) = true 

ac/s out (p,g,i) 
Precondition: 

s'.pkt-from(q,i) = true 

a .ac/;_je7it(g, i) = false 
Effect: 

s.acfc_sent(g, i) = true 

ok(p,i) 

Precondition: 

s' .client .output -recd(i) = true 
Vg £ neighbor s(p) , 
if s' .pkt-for(q, i) = true 
then s'.acfe_rec<i(g, i) = true; 
s .ok^sent(i) = false 
Effect: 

s.okjent(i) = true 

9°{P, «) 
Effect: 

s.gojrecd(i) = true 

client J,nput(jp, M, i) 
Precondition: 

s .gojrecd(i) = true 

M = s' .inbox(i) 

s' .client Jnputsent(i) = false 
Effect: 

s .client -.inputsent{i) = true 
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Partitions 

All client -input(p, *, *) actions in one class, 
all send out (p, q, *, *) actions in one class, for each q £ neigh(p), 
all ack out (p, q, *, *) actions in one class, for each q £ neigh(p), 
all ok(p, *) actions in one class. 

2.2.2 Link automata 

Front-end automata exchange messages through link automata that model communication channels 
between nodes in the network. We model an edge between any two nodes p and q of the network 
with two unidirectional link automata L(p ->■ q) and L(q — *■ p) which, which carry messages from 
one front-end to another. We specify L(p — ^ q) below. 

Input: 

send out (p, q, fi, i), \i a set of messages 
ack° ut (p, q, i) 

Output: 

send tnp (p, q, /*,»), n a set of messages 
ack inp (p,q,i) 

Internal: 
none 

State 

buffer, a first-in first-out queue of elements of arbitrary type, initially empty 
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Transitions 



send° ut {p,q,n,i) 
Effect: 

Add aend(n, i) to buffer 

ac/s < ' ut (p,g,i) 
Effect: 

Add ack(i) to buffer 

8end'" p (p,q,n,i) 
Precondition: 

first(»'.6u//er) — send(n,i) 
Effect: 

s.buffer = rest(s .buffer) 

ack in "(p,q,i) 
Precondition: 

first(s'.6u//er) = ack(i) 
Effect: 

s.buffer = rest(s'. buffer) 



Partitions 

All send out (p, q, *, *) actions in one class, 
all ack out (p, q, *) actions in one class. 



2.2.3 The local synchronizer 

As mentioned earlier, locjsynch ensures that a client's neighbors have all executed a round of the 
synchronous algorithm before allowing it to begin the next round. 

Input : 

ok(p,i),p £ V 

Output : 

go(p,i),p € V 
Internal: 



State 



array gosent(p, i), pg V 
array ok-recd(p, i), p £ V 
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Transitions 



ok(p,i) 
Effect: 

s.ok-recd(p,i) = true 

9°(p, i) 

Precondition: 

Vg such that q 6 neigh(p) U {p}, s'.ok-recd(q,i) = true 

s .gosent(p, i) — false 
Effect: 

s.gosent(p, i) = true 



Partitions 

All go(p, *) actions in one class, for each p £ V. 

In the upcoming section, we will show how every locally synchronous execution of a synchronous 
algorithm corresponds to a globally synchronous execution with respect to any client. 

2.3 Proof of implementation 

Let LF be the automaton formed by the composition of the Local synchronizer with all the 
Front-end(j>) automata and the links in between them, and LC — LF ■Jl p ^vClient(p), be the "Lo- 
cal synchronizer and Client automata" composition. As mentioned earlier, automaton globjsynch 
differs from its "implementation" LF in that it begins a new round of communication with its 
clients only after they have all replied to the earlier set of messages. In the case of LF, on the 
other hand, locsynch allows a client to proceed to a subsequent round as soon as the client itself 
and all its neighbors have responded to the earlier rounds' messages. In this section we will argue 
that this divergence in functionality is acceptable by proving that LF "implements" globjsynch in 
a precisely defined, but unconventional sense. Put in informal terms, what we will show is that as 
far as each client can tell, it will be running in a synchronous system. 

Let GC ("Global synchronizer and Client composition") be the composition of globjsynch with 
all the client automaton. Our task will be to show that given any behavior /3 of LC, there exists a 
behavior 7 of GC such that /3 and 7 "look alike" to each of the clients. That is: 

Theorem Given a behavior (3 of LC, there exists 7 £ behs(GC) such that for all p £ V, 
P\Client(p) = <y\Client(p). 

To this end, we will reorder the sequence of actions in /3 to get /3' such that (3' under an 
appropriate projection gives 7, a behavior of GC with the required property. 

To construct the required permutation /3' of /3, we will define a relation that captures a depen- 
dency notion among the actions in /3, and will prove that any permutation of these actions that 
is consistent with the relation is also a behavior of LC. We will then prove that it is possible to 
transform any /3 into a sequence of actions /3', while preserving the constraints imposed by the 
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relation, so that /3' mirrors the way GC interacts with its clients (with outgoing messages to clients 
being sent only after each client has responded to all previous communications). From the fact 
that j5' is also a behavior of LC, we will be able to show that its projection 7 on acts(GC) yields 
a behavior of the latter automaton, and that /3 and 7 project identically on each of the clients. 

After proving that LC thus "implements" GC, we will justify why executions of GC thus 
generated are fair to its component automata, thus showing that LC results in fair implementations 
of GC. 

2.3.1 Basic properties of the locally synchronous system 

Before we state the dependency relation, we will prove some results that will allow us to reason 
about LF's working. 

Each client's proper behavior is predicated upon its receiving appropriate inputs in response to 
its actions. We proceed to show that LF preserves well-formedness so that we are guaranteed that 
every client's behavior is also a well-formed sequence of actions. 

Theorem 2.1 For all p G V, LF preserves p-well-formedness. That is, letting $ p — ext(client(p)) , 
for all p G V, if /3|* p G W p , and n G out(LF) is such that (3iv\ext(LF) G finbehs(LF), then 

Ptv\$ p G W p . 

Proof: For all p, if n £ $ p , or /3\$ p is the empty sequence of actions, the theorem holds trivially. 
If not, n = client Jnput(p, M, i) for some M and i. Since <J> p C acts(LF) and (3n\ext(LF) is finite, 
/3|3> p is finite too, and has a last action. 
Let p be that last action in /3|$ p . We need to show that p = client joutput{p, M' , i) for some M'. 

As mentioned in the section on Preliminaries, all the component automata of LF are deter- 
ministic and none of the automata whose composition is LF has internal actions. The projection 
(3\ext(LF) G finbehs(LF) thus determines a unique execution of LF, and hence a unique state 
(say s') that LF will reach after that execution. Since flir\acts(LF) G behs(LF), we have, from 
the code implementing the client-input action in front-end(p), s'[front-end(p)].gojrecd(i) = true. 
The state variable gojrecd(i) is set to true iff go(p, i) is received from locjsynch, and hence we 
examine the preconditions governing the execution of that action, in locjsynch. 

One precondition ensures that s'\locjsynch].okjrecd{p,i) = true. (We exploit our implementa- 
tional convention that boolean values, once set to true in an execution, remain so for the remainder 
of that execution). This, in turn, implies that in front-end(p), s'[front-end(p)].ok^sent(i) — 
true, which is predicated upon s'[front-end(p)].client .output jrecd(i) being true. The action 
client ..output (p, *, i) therefore occurs in j3. 

Action client J.nput{p, *, i)'s preconditions imply that it has not occurred in /3 if fln\acts(LF) 
occurs in finbehs(LF). We can now infer that p = clientjoutput{p,*,i). For, if not, let p be 
the action immediately following client joutput{p, *, i) in /3|$ p . Since /3|3> p G W p , p' is of the form 
client Jnput(p, *,i), which is a contradiction. B 

Another useful property of LC is that actions are never repeated in its behaviors. We prove 
that now. 

Theorem 2.2 In any /3 G behs(LC) , for any p, q and i , each of ack mp (q,p,i),ack out (p,q,i),ok(p,i) 
and go(p, i) occurs at most once in (3. Furthermore, for any p, q and i, there can be at most one 
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action of each of the following forms in /3 : 

client -output(p, *, i), send out (p, q, *, i), send inp (p, q, *, i), clientJnput(p, *, i) . 

Proof: 

Each client-Output(p,*,i), for any p and i, is the output action of a client that, by defini- 
tion, preserves well-formedness. It is thus guaranteed that i (which we interpret to be the round 
number) takes on increasing values with successive occurrences of the action, and hence that 
client -Output(p, *, i) occurs at most once. 

Every other action in j3 is a locally controlled action of one of the component automata of LC 
(since no component automaton uses internal actions). The "code" implementing each ensures 
non-repetition, as can be verified by inspection. ■ 

We will now describe the dependency relation. With each schedule of LC there is a relation — > p 
that captures the ordering that LC imposes upon the actions in /3. The key property of this relation 
is that for any schedule f3 of the implementation system, and any permutation /3' of /3 that preserves 
the partial order of events given by — } p , we have that /3' is also a schedule of the implementation 
system. In other words, — > p captures enough about the dependencies in the schedule to ensure that 
any reordering that is consistent with these dependencies is still a valid schedule of the system. 

We give a formal definition of the relation — ^. 

Definition 2.2 Given a schedule /3 of LC and two actions tv and p that occur in (3, the pair (n,p) 
belongs to the relation —} p (we write tt -} p p) if n and p have the form of one of the following pairs, 
for any p £ V, for any q £ neigh(p), for any q' £ neigh(p) U {p}, for any sets of tagged messages 
M and M', for any set of messages p., and for any i > 0: 

client jinput(p, M', i) and client -Output(p, M, i + 1), 

client-Output(p, M, i) and send out (p, q, p, i), 

send out (p, q, p, i) and sen<r np (p, q, p, i), 

send mp (p, q, p, i) and ack out (q, p, i), 

ack out (q 1 p, i) and ack inp (q,p, i), 

ack tnp (q,p, i) and ok(p, i), 

client-Output(p, M, i) and ok(p, i), 

ok(q',i) and go(p,i) or 
go(p, i) and client Jnput(p, M, i) . 

We write p\ -+ p pi -^ p ■ • • -^ p p r to mean pi -^ p pi + i for alH, < i < r 

Definition 2.3 Given any behavior j3 of LC, we write n ^> p p to imply that actions 7r and p occur 
in that order in fl, though not necessarily consecutively. 

Remark: Note that since Theorem 2.2 ensures that there are no repeated occurrences of any action, 
for any distinct actiona tv and p that occur in /3, exactly one of tv ^> p p and p ^> p n will be true. 

In lemmas to follow, we will use the definitions given above and associate the dependency 
relation with the relative order of occurrence of actions within a schedule. We will first show that 
any for two actions it and p of a schedule /3 of LF such that w -+ p p, it will also be the case that 
7T-^/3 p. 
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Definition 2.4 A permutation f3' of (3 E behs(LC) is said to be consistent with the relation -^ p 
(—> ^-consistent) if for any distinct actions n and p in (3, n ~~»$i p whenever n — ^ p. 

We will then use these lemmas to show that given any permutation of a schedule of the imple- 
mentation automaton continues to remain a schedule if the relative ordering of actions is consistent 
with the dependency relation. 

Lemma 2.3 Let fl £ behs(LC). For any p, q, p, M and i of the right type, 

1. If client -Output(p, M,i + 1) occurs in (3 then there exists a set M' of tagged messages such 
that client Jnput(p, M', i) ~>^ client -Output(p, M, i + 1). 

2. If send° ut (p,q, p,i) occurs in f3 then there exists a set N of tagged messages such that 
{(m, q)\m £ p} C N AND client^output(p, N, i) ~^ send out (p, q, p, i). 

3. If send mp (p, q, p, i) occurs in (3 then send out (p, q, p, i) ~> /3 send mp (p, q, p, i). 

4- Ifack out (p, q, i) occurs in/3 then there exists a set v of messages such that send inp (q,p, v, i) "~>p 
ack out {p,q,i). 

5. If ack tnp (p, q, i) occurs in [3 then ack out (p, q, i) -^>p ack mp (p, q, i) . 

6. If ok(q,i) occurs in f3 then, for all p £ neigh(q) for which there exists a set u of messages 
such that send out (q,p, u, i) occurs in (3, it will be the case that ack inp (p, q, i) ^> p ok(q, i). Also, 
there exists a set N of tagged messages such that client-Output(p, N, i) ^>$ ok(q, i). 

7. If go(p, i) occurs in (3 then Mq such that q £ neigh(p) U {p}, ok(q, i) ^> go(p, i). 

8. If client J.nput(p, M, i) occurs in (3 then go(p, i) ^>p client Jnput(p, M, i). 

Proof: 

1. This follows directly from the fact that LF and all the clients preserve W p , and hence 
Vp, f3\client(p) £ W p . 

2. Verified by inspecting the code implementing actions client -output and send ™ 1 in front -end{p) 

3. send inp (p,q, p,i) is an output action of L(p — *■ q). In order for this action to be enabled, 
send out (p, q, p, i) must have occurred at an earlier point in (3. 

The other cases are verified similarly. H 

The main goal of this subsection is to show that any permutation of (3 £ behs(LC) that is 
consistent with — > p is also a behavior of LC. To this end, we will need the lemmas given below. 

Definition 2.5 The transitive closure of the -+p relation (written *-^p) is defined the usual way. 

Thus tv !-»£ a implies 3pi i = 1, • ■ ■ , r, r > 1, such that pi = 7r, p T = a and pi — > p p% —¥ p • • ■ — ^ p r . 
Consistency of a schedule with respect to 1-^ is defined in the way — ^-consistency is, and we 
state the equivalence of the two notions in this lemma: 
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Lemma 2.4 Every — ^- consistent permutation of /3 £ behs(LC) is i— » ^-consistent. 

Proof: By induction. B 

Lemma 2.5 Every behavior /3 of LC is —tp- consistent. 

Proof: Follows from Lemma 2.3.1-8. For example, let us verify that Vp £ V", V<7 6 neigh(p), 
\i ack inp (q,p,i) — ^ ofc(p, i), then in (3,ack inp (q,p,i) ^>p ofc(p, i). Since ack mp (q,p, i) — ^ ofc(p, i), 
ac& mp (<7,p, i) occurs in /3. By successively applying Lemma 2.3.5 and Lemma 2.3.4, we get 3v such 
that send znp (p, q, v, i) occurs in j3. Then by Lemma 2.3.3, send out (p, q, v, i) occurs in /3. Thus, by 
Lemma 2.3.6, we get ack inp (q,p, i) ~~*p ok(p, i). Other cases follows similarly. H 

Corollary 2.6 Every behavior f3 of LC is >->^- consistent. 

We will use these results to to prove the main theorem of this subsection: 

Theorem 2.7 Given (3 £ behs(LC), let /3' be any permutation of fl consistent with — >- /9 . Then 
p' E behs(LC). 

Proof: To prove that /?' £ behs(LC), it suffices to show that when /3' is projected on each 
component automaton of LC, the resulting sequences are behaviors of executions of the respective 
automaton [3]. We will provide the proof in the case when the component automaton is a front-end, 
and omit the easier proofs that establish the result for the other component automata of LC. 

Our proof will be based, in part, on the following fact: 
Proposition Let /3 be any schedule of LC. If send inp (q,p, ft, i) and client Jnput(p, M, i) occur in 
(3, then 

seraf np (<7, p, /z, i) t-^-p client -input (p, M, i). 

Proof: We successively apply sub-lemmas 2.3.8 and 2.3.7, in that order, instantiating quantifiers 
as the notation suggests. From sub-lemma 2.3.3, send out (q,p, /z, i) occurs in /3 and hence from 2.3.6, 
ack inp (p, q, i) occurs in j3. Applying 2.3.5 and 2.3.4, we find that there exists an v such that actions 
send mp (q,p,v,i), ack out (p,q,i), ack inp (p,q,i), ok(q,i), go(p,i), and client Jnput(p, M,i) all occur 
in /3. By Theorem 2.2 v = fi. Since each successive pair of actions in the above sequence is related 
by — >p, send inp (q, p, fi, i) ^p client Jnput(p, M, i). B 

Lemma 2.8 Vp £ V, /3'\front-end(p) £ behs(front-end(p)). 

Proof of lemma: The proof is by induction on the length of /?'. In the case when |/3'| = 0, the theo- 
rem holds trivially. Otherwise let S be a prefix of /3' such that 8\front-end(p) £ behs(frontjend{p)). 
Consider 7 = Sn, where n £ acts(front-end(p)) is the next action in f3' (We need to look only at 
such actions). 

If 7r is an input action, 7 £ behs(front-end(p)), as inputs are always enabled. If not, n is 
one of client Jnput(p, M, i), send out {p, q, /x, f), ack out (p, q, i) or ok(p, i), for some q, /z, M and i. We 
need to show that in each case, that particular output action must be enabled. A proof for the 
case when tv — client Jnput(p, M, i) follows. If n — client Jnput(p, M, i), by Lemma 2.3.8, go(p, i) 
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occurs before it in /3. Since /3' is consistent with — ^ the same order of occurrence is maintained 
in 7. By Theorem 2.2 we know that no action of the form client Jnput(p, *, i) has occurred at an 
earlier point in 7. We must now show that M will have the correct value. In other words, we need 
this claim: 
Claim Vq, fi such that send inp (q,p, fi, i) occurs in /3, 

send inp (q,p, fi, i) ^> p client Jnput(p, M, i) iff send inp (q,p, fi, i) ~> 7 client lnput(p, M, i). 

Proof of claim: 

(-►) 

By the Proposition and Lemma 2.4. 

{ *- ] m 

By Corollary 2.6 and the Proposition. ■ 

Hence the preconditions for the client Jnput(p, M, i) output action of automaton front.end{p) have 
been satisfied, and the action is currently enabled, i.e. jn £ behs(front_end(p)). 
When 7r is send out {p,q,fi,i) for some fi, or ack out (p,q,i) or ok(p,i), the fact that n is enabled 
follows immediately from the appropriate case of Lemma 2.3. ^ 

Similar proofs for each of the other component automata in LC show that (3' projects correctly on 
each. Hence j3' £ behs(LC). * 

2.3.2 Reordering yields synchronized behaviors 

We will now specify a way of reordering the actions of any behavior f3 of LC while maintaining 
->p such that the resulting j3' £ behs(LC) will "look like" a behavior of GC. In particular, the 
reordered schedule will correspond to executions of LF where, at every round, front-ends send out 
inputs to clients only after they have confirmed that all the tagged messages in every client's output 
have been delivered to their destinations. 

We shall call such behaviors "synchronized", and show how to construct the synchronized 
counterpart of any behavior of LC. In the next subsection, we will prove that upon projection 
onto GC, any synchronized behavior of LC will result in a behavior of GC. We will then show 
that given any behavior f3 of LC, no client will be able to distinguish j3 from the behavior of GC 
obtained by projecting onto GC the synchronized counterpart of /3. 

Definition 2.6 A behavior (3 £ behs(LC) is said to be a synchronized behavior if for all i, and for 
all p, q such that actions ok(p, i) and go(q, i) are in (3, ok{p, i) ~>^ go(q, i). 

Definition 2.7 Given {3 £ behs(LC), let 

• actions^, i) — {tv occurs in f3\n is tagged with round number i}. 

• client Jnputs(/3, i) = {tt\3M,p such that n = client Jnput(p, M, i) occurs in (3}. 

• ofcs(/3, i) = {tt\ for some p, n = ok(p, i)}, 

• gos{f3, i) = {71- 1 for some p, n = go(p, i)}. 

• other _acis(/3,z) = {7r|7r occurs in j3 and is not in client Jnputs(/3,i), oks(/3,i) or gos(f3,i)} 

21 



We now provide a way of extracting from a behavior /3 of LC, a synchronized behavior of LC. 

Definition 2.8 If fr, i = 1, 2, • • •, n are finite sequences of actions, let 0" =1 {/3i} be the sequence 
concatenation (ixfii- ■ -fi n - 

Theorem 2.9 For any behavior f3 G behs(LC), consider 

oo 

(3' = Q){f3\other-acts(P, i) ■ /3|ofcs(/3, i) ■ /3\gos(f3, i) ■ (3\clientJnputs{(3, i)}. 

f3' is a synchronized behavior of LC. 

Proof: (3' is a permutation of (3 as every action of LC is tagged with a round number and so 
is an element of exactly one actions(f3, i) for some i, and other _acis(/3, i), oks((3, i), gos(f3, i) and 
client-inputs(/3, i) partition actions^, i) for each i. 

We need to show that (3' is consistent with the -^ relation. First, we observe that by Lemma 
2.5, /3 itself is consistent with — > 3 relation. 

Consider relations involving actions with the same round number, say i. It can be checked that 
the above reordering violates none of the — ^-consistency requirements, as none of the relations 
involving actions ok, go and client Anput are violated. 

Turning to orderings involving actions with different round numbers, all we have to show is that 
orderings of the form client Anput(p, *, i) -¥ p client joutput{p, *, i + 1) are maintained. But this is 
true by construction. 

f3' is thus consistent with the -^ . Hence, by Theorem 2.7, /3' G behs(LC). /3' is synchronized by 
construction, and this completes the proof. ^ 

2.3.3 Synchronized behaviors are globally synchronous 

Given /3 £ behs(LC), let /3' be the synchronized behavior obtained after applying Theorem 2.9 to 
(3. We will now extract from /3' a behavior 7 of GC which is such that with respect to each client 
automaton, 7 and /3 are indistinguishable. 

We will take 7 to be f3'\acts{GC), and first show that 

7 = /3'\acts(GC) G behs(GC). 

We consider a restriction of automaton locsynch that will enforce synchronized behavior. We will 
show that the composition of the front-ends and the restricted locsynch automaton will generate all 
the synchronized behaviors of LC. This will be followed by a straightforward possibilities mapping 
proof showing that this composition implements GC. We will thus have proved that 7 is a behavior 
of GC. We will then conclude with the proof that no client can tell 7 and j3 apart. 

Theorem 2.10 If f3' G behs(LC) is synchronized then f3'\acts{GC) G behs(GC). 

Proof: Consider a restriction of locjsynch to Rlocsynch, where automaton Rlocsynch has the 
same signature and states as locjsynch, and executes the same transitions that locjsynch does on 
all but one of locjsynch actions. The automaton Rloc^synch will differ from locsynch only in 
its implementation of the action go(*, *). The restricted version of the locjsynch automaton will 
ensure that all front-ends are "safe" before it gives any front-end permission to send its client that 
rounds' input. 
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Rloc_synch 

Input : 

ok(p,i),p £ V 

Output: 

go(p,i),p £ V 
Internal: 



State 



array gosent(p, i), p£V 
array okjrecd(p, i), p G V 



Transitions 



ok(p,i) 
Effect: 

s.ok-recd(p, i) — true 

go(p, i) 

Precondition: 

for all q £ V, s' .okjrecd(q, i) = true 

s ,gosent(p, i) = false 
Effect: 

s.gosent(p, i) = true 



Partitions 

All go(p, *) actions in one class, for each p EV. 



Let 

RLC — Rlocjsynch • JJ front.end(p) ■ client(p) 

Then the following claim is supported by a direct proof: 

Claim Every synchronized behavior of LC is a behavior of RLC. 

Proof of claim: Indeed, a behavior f3 of LC may diverge from being a behavior of RLC only at an 
occurrence of go(*, *) as it is this action alone that needs a different (stronger) set of preconditions 
satisfied in RLC. But it is this very case that is taken care of in the requirement that (3 be 
synchronized. Hence the proof follows. B 

The theorem will stand proved if we show that RLC implements GC. 
Let S = {tt g acts(RLC) : tv <£ acts(GC)} and RLC = hide^RLC. 



Lemma 2.11 Consider f defined as follows: Vs G states(RLC) , 
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f(s) = {t G states(GC) \ for all p and i, 

t.clientjoutput-recd(p,i) = s[front-.end(p)].clientjoutput-recd(i) 
t.clientJnputsent(p,i) = s[front-end(p)]. client Input sent{i) 
t.tray(p,i) = \J reneigh{p) {(m,r) | s[front.end(r)].pkt.for{p, i) = true and 

m G s[front-end(r)].outbox(p, i)}. 



Then f is a possibilites mapping from RLC to GC . 

The above proof will need the two claims that follow. The truth of these claims are established by 
operational means, and are omitted here. 

Claim 2.1 For s G states(RLC), if s.[front„end(p)].gojrecd(i) = true for some p G V , then 

1. s.[front-end(q)].clientjmtput-.recd(i) = true for all q EV . 

2. s.[front-end(p)].inbox(i) = \J r£neigh ( p ) {(m,r) | s[front.end(r)].pkt-for(p, i) = true and 

m G s[front-end(r)].outbox(p, i)}. 

Proof of lemma: If s G start(RLC) then the existence of a start state in GC consistent with 
/ is easily verified. Let s' be a reachable state of RLC, t' any state in f(s') and (s 1 , n, s) a 
step of RLC. We need to produce an extended step of GC (f, 7, t) such that t G /(&) and 
j\ext(LS) = n\ext(RLC). We handle cases differently depending on the kind of action n is. 

When 7r is a hidden action, that is, n G S, then 7 will clearly have to be a sequence of no (zero) 
actions for / to be obeyed. It can be seen that no hidden action changes the values of the variables 
that determine the action of /. Hence we can take t = t'. 

Otherwise, if tt is an input action of RLC and GC, meaning that n = client. output(p, M, i) for 
some p, M and i, we let 7 = tt. By inspection, it can be seen that the final state t of GC will be 
in /(*). 

If not, 7r = client Jnput{p,M,i) for some p, M and i. Again, taking 7 = ir and then using 
Claim 2.1, we can see that the preconditions of it are satisfied in GC if they are met in RLC. 
Further, the occurrence of this action sets the client -input sent variable to true thus ensuring the 
continued consistency of /. 

Thus / is a possibilities mapping, and behs(RLC) C behs(GC). B 

We thus prove the main theorem of this section. 

Theorem 2.12 Let {3 be any behavior of LC. Then there exists 7 G behs(GC) such that for all 
peV, P\Client(p) - j\Client(p). 

Proof: By Theorem 2.9 and Theorem 2.10 there exists /3', a permutation of /3 such that 7 = 
/3'\acts(GC) G behs(GC). Since both /3 and 7 are consistent with -^, the relative ordering of all 
client -Output(jp, *, *) and client Jnput(p, *, *) actions in /3 is maintained in 7 for any p in V, and 
since 7 is a projection of a permutation of /3, the contents of the tagged message sets exchanged 
with the clients is not altered. Hence the two sequences project identically on each client. I 

We will now examine LC and sketch a proof of fairness of the implementation. 
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2.4 Fairness of implementation 

We have shown that behaviors of LC generate behaviors of GC. Since globsynch is deterministic 
in its functioning, such behaviors yield unique executions of that automaton. We will now argue 
why such executions obtained from fair behaviors of LC are fair executions of globsynch. 

Let f3 be any fair behavior of LC, and let 7 be a behavior of GC which projects as f3 does on 
all the clients, as in Theorem 2.12. We assume also that 7 is constructed as in that theorem, by 
considering behavior /3' of RLC, and let Y be any execution of GC that has 7 as its behavior (Note 
that since the clients have internal actions that we have no information about, distinct executions 
of GC may yield indistinguishable behaviors). If T is not a fair execution, it can be shown that 
/3' will not be a fair behavior of RLC. This will imply that f3 is not a fair behavior of LC, thus 
deriving a contradiction. 

Combining the fairness argument with Theorem 2.12, we then have 

Theorem 2.13 Let f3 be any fair behavior of LC . Then there exists 7 G fairbehs(GC) such that 
for all p G V , (3\Client(p) = f\Client(p). 

We will first show that the global synchronizer preserves the well-formedness property, and 
then, using this fact, prove that if the behavior 7 of GC generated is unfair, then the behavior /3' 
of RLC that 7 was "extracted" from, is also unfair. 

Theorem 2.14 Automaton globsynch preserves p-well-formedness, for all p G V . That is, for 
each p in V, letting $ p = ext(client(p)) , »//3|* p G W p , and it G out{glob synch) is such that 
f3n\ext(glob synch) G finbehs(globsynch), then f3ir\$ p G W p . 

Proof: The proof of this theorem resembles that of Theorem 2.1 strongly, 

For each p G V, if n £ $ p or f3\$ p is the empty sequence of actions, the theorem holds trivially. 

If not, 7r = client Jnput(p, M, i) for some M and i. Since $ p C acts(LF) and f3n\ext(LF) is finite, 

/3|$ p is finite too, and has a last action. 

Let p be that last action in /3|* p . We need to show that p = client joutput{p, M' , i) for some M'. 
Since globsynch is a deterministic I/O automaton, and has no internal actions, the projection 

fi\ext{glob synch) thus determines a unique execution, and hence a unique state (say s') that 

globsynch reaches after that execution. Since f3n\ext(glob synch) G behs(globsynch), we have, 

from the code implementing globsynch, s'[globsynch}.clientjoutputjrecd{p,i) — true. Thus the 

action client joutput(p, *, i) occurs in /3. 

Action client Jnput(p, *, i)'s preconditions imply that it has not occurred in /3 if (3n\acts(LF) 

occurs in finbehs(LF). We can now infer that p = client joutput(p,*,i). For, if not, let p' be 

the action immediately following client joutput{p, *, i) in /3|3> p . Since /3|<& p G W p , p' is of the form 

client Jnput(p, *, i), which is a contradiction. 

■ 

We now proceed to the theorems that establishes the fairness properties of the implementation. 

Let /3' be a behavior of automaton RLC, extracted from a behavior /3 of LC as per Theorem 2.9. 
Behavior (3' is a synchronized, as can be checked from the code implementing RLC. By Theorem 
2.10, 7 = P'\acts(GC) is a behavior of GC. 

Theorem 2.15 If ft' is a fair behavior of RLC, then 7 is a fair behavior of GC . 
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Proof: 

If not, let 7 be an unfair behavior of GC, and p £ V be such that actions from the class of 
actions of the form client Jnput(p, *, *) are enabled but never performed in 7. Then there will be 
a smallest i such that client Jnput{p,*,i) is enabled, but never performed in T (an execution of 
GC whose behavior is 7) because once an action of that form is enabled, it is not disabled unless 
it is performed. It follows that client J.nput{p,*,i) does not occur in f3'. Since the automaton 
globjsynch preserves well-formedness with respect to all the clients, and the clients preserve well- 
formedness, we hence have that client joutput{p, *, i + 1) will not occur in T, and hence not in /3'. 
From the code implementing the automaton RLC, this in turn implies that no action of the form 
client Anput{*, *, i + 1) will occur in /3' (the intuitive reason is that RLC has been built to permit 
front-ends to respond with a round's messages to the client only if all the clients have sent in their 
messages for that round), thus putting an end to the exchanges between any front-end and its 
client. Thus if 7 is an unfair behavior of GC, then 7 is finite. 

We consider the case when 7 is finite. If a finite behavior 7 is unfair, then it follows that in the 
final state t of T, an action of the form client -input (p, *, i) is enabled, for some p EV . 

Let a' be an execution that has /3' as its behavior. We know that in GC, 
t. client .input sent{p,i) is false, and hence that in front-end(p), the corresponding 
client .input. sent variable is false throughout that execution. In order that a' not have enabled 
actions in the final state s of automaton RLC, it follows that s[front-end(p)].gojrecd(i) has to be 
false. In turn, in order that locsynch not have the go action enabled in s, s[locsynch].okjrecd(q, i) - 
false for some q £ V. Working backwards in like fashion, we arrive at the fact that for some 
r 6 neigh(q), s[front.end(q)].pkt.for(r,i) = true, but s[front-end(q)].ackjrecd(r, i) — false. 
This, however, can be seen to be to imply that a' is not a fair execution of automaton RLC, thus 
completing the proof. ^ 

Thus we have proved that if 7 is unfair, so is /3'. We now proceed to show that if /3' is unfair, /3 
(the behavior of LC that was permuted to yield (3') is not fair, thereby proving that fair behaviors 
of LC correspond to fair behaviors of GC, as far as each client can tell. 

As before, let behavior /3' of RLC correspond to an execution a' of RLC, and f3 G behs(LC) 
correspond to an execution a of LC. We will show that if an action is "permanently enabled" in 
a', then the same is true of that action in a. 

Theorem 2.16 If a is a fair execution of LC, then a' is a fair execution of RLC. 

Proof: If a' is an unfair execution of RLC, then there exists a class of actions C, and a state s in 
a' such that in s and all subsequent states (if any), some action from that class is always enabled. 
We first consider the case when C is the class of actions of the form client Jnput(p, *, *) for some 
p in V. There will be a smallest i such that client Jnput(p, *, i) is enabled in state s of a' and all 
subsequent states, but never performed. Thus client Jnput(p, *, i) never occurs in a', and hence not 
in a as well. Since that action is enabled, it follows that s[front-end(p)].gojrecd(i) is true, meaning 
that go(p,i) occurs in a' and a. Thus in a, the action go(p,i) occurs, while clientJ,nput(p,*,i) 
never does. But this enables the latter action permanently, thus making a unfair. We have thus 
shown that if a' is an execution unfair to this choice of C, so is a. 
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Similar proofs apply when C is a class consisting of actions of any of the other possible forms: 
send inp , send out , ack inp , ack™*, ok, go or client Jnput. This completes the proof. ■ 

Thus we have shown that from the point of view of each client, fair behaviors of the implemen- 
tation automata are indistinguishable from fair behaviors of the global synchronizer. 

The rest of this report will deal with fashioning a provably correct fair distributed implemenation 
of the local synchronizer locjsynch. 

3 Implementing a Network Synchronizer 

In this section, our task will be to provide a fair, distributed implementation of the automaton 
locsynch described in Section 2.2. Our reasons for doing so are straightforward. From Theorem 
2.13 of the earlier section, we know that fair behaviors of clients and front-ends interacting with 
locsynch "look like" fairs behaviors of GC to each client. A distributed and fair node- level imple- 
mentation of automaton locsynch will thus leave us with a synchronizing system whose behaviors 
are indistinguishable from those of GC, as far as each client is concerned. 

The automaton locsynch is implemented in two stages. We partition the graph into clusters of 
node, and assume, in the first stage, that we have some means of synchronizing all the nodes in each 
cluster. We then describe the process of synchronizing all the clusters. In the second stage of our 
implementation, we consider the problem of synchronizing the nodes inside each cluster (we provide 
node-level implementations) and the problem of giving node-level implementions of the intercluster 
synchronization process of the earlier stage. We will deal with fairness issues in each stage, thus 
ensuring that, at the end of the section, we have a fair and correct node-level implementation of a 
network synchronozer. 

3.1 Cluster Level Synchronization 

In this section we will describe a set of automata that will implement locsynch. 

The graph is divided into clusters of nodes, each with its own spanning tree (we assume that this 
spanning forest is constructed with some preprocessing). The synchronizing algorithm is derived as 
a composition of a simple synchronizer (called /3 in [1]) executing within each cluster, and another 
simple synchronizer (called a) that synchronizes between the clusters. 

Each cluster implements a distributed algorithm that interfaces with the front-end of each 
client automaton associated with each node in the cluster. At each pulse, this algorithm, an intra- 
cluster synchronizer, receives and exchanges messages and detects the condition that all messages 
of the underlying synchronous algorithm for that pulse sent out by each node in the cluster have 
been received. The algorithm now passes this information on to an intercluster synchronizer. 

When the intercluster synchronizer receives this information from all neighboring clusters and 
the cluster in question, it gives the intracluster synchronizer permission to allow nodes in its cluster 
to execute the next pulse of the synchronous algorithm. Note that this behavior closely parallels 
that of locsynch, the automaton that enforces locally synchronous behavior between neighboring 
clients. A pictorial representation of the cluster level implementation follows in figure 3. 

We begin by specifying the implementation automata that will carry out the above tasks, and 
subsequently prove the correctness of the implementation using mapping techniques. We will then 
briefly argue that fair behaviors of the implementation will given rise to fair behaviors of the 
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Figure 3: Automata implementing the local synchronizer 

implemented system. In later sections, we will give distributed implementations of the intracluster 
and intercluster synchronizers described here. 

We call "forest" the spanning forest that is assumed to be given. If C is a subtree of forest, 
neighboring JLrees(C) is the set of trees in forest that edges from nodes in C are incident upon, 
and nodes(C) is the set of nodes in C. With each subtree C of forest we associate an automaton, 
an intracluster synchronizer (cluster synch(C)) that does the job of synchronizing the clients on 
the nodes in the cluster. We give the formal specifications of an arbitrary intracluster synchronizer 
below. 

3.1.1 Cluster level synchronizer 

Automaton cluster synch(C), C a subtree of forest: 



Signature 

Input : 

ok(p,i),p 6 nodes(C) 
cluster -go(C , i) 

Output: 

go(p, i),p £ nodes(C) 
cluster jok(C , i) 

Internal: 
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State 



array gosent(p, i), p £ nodes(C) 
array ok-recd(p, i), p £ nodes(C) 
array cluster -go jrecd(i) 
array cluster joksent(i) 



Transitions 



ok(p,i) 
Effect: 

s.ok-recd(p,i) = true 

cluster jok(C,i) 

Precondition: 

Vp G no<2es(C),s'.o/s_rec<i(p, i) — true 
s.clusterjoksent[i) — false 

Effect: 

s .cluster -oksent(i) = true 

cluster ~go(C \ i) 
Effect: 

s. cluster -go -.recd(i) = true 

go(p, i) 

Precondition: 

s' .cluster -gojrecd{i) = true 

s'.gosent(p,i) = false 
Effect: 

s.gosent(p, i) = true 



Partitions 

All go(p, *) actions in one class, for each p £ nodes{C) 
all cluster -go(C, *) actions in one class. 

3.1.2 A synchronizer for clusters 

The intercluster synchronization part of the implementation is carried out by an automata called 
forestsynch, as described earlier. As mentioned earlier, its functioning, captured in the formalism 
given below, resembles that of LS closely. 

Input: 

cluster jok(C ', i), C a subtree of forest 
Output: 

cluster -go[C,i),C a subtree of forest 
Internal: 

none 
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State 

array cluster jokjrecd{C, i) for all C a subtree of forest 
array cluster. go.sent(C, i) , for all C a subtree of forest 



Transitions 



cluster .ok(C ', i) 
Effect: 

s. cluster ,x)k-recd(C, i) = true 

cluster .go(C, i) 
Precondition: 

VC' £ neighboring -trees(C) U {C}, 

a'.c/u«ter_ofc_recd(C',i) = true 
cluster. go.sent(C, i) — false 
Effect: 

cluster. go.sent(C, i) = true 



Partitions 

All cluster. go(C, *) actions in one class, for each C a subtree of forest. 

3.2 Proof of implementation 

We will now prove that the compostion of forestsynch and the cluster synch automata (call it 
CF) implements locsynch, in the sense involving behavioral inclusion. The possiblities mapping 
proof we provide will show that every behavior of CF, when projected onto locsynch, will yield a 
behavior of the latter automaton. 

We first state a lemma whose truth follows directly from definitions. 

Lemma 3.1 For all p, C such that p £ nodes(C), 

{p} U neigh(p) C {q\ q £ nodes (C) such that C G neighboring JLrees{C) U {C}}. 

Let S = {tv E acts(CF) : n = cluster-go(*, *) or cluster _ofc(* , *)}, and CF = hide^CF. We 
will need the following lemmas for the main theorem of this section that guarantees implementation. 

Lemma 3.2 For any p £ V, let C be the cluster such that p £ node{C). 

If CF is in state s such that s[clustersynch(C)].cluster.go-recd(i) — true then 

VC" £ neighboring Jtrees{C) U {C}, s[forest^synch].cluster-ok-recd(C, i) = true. 

Lemma 3.3 For any C a subtree of forest, if CF is in state s such that 
if s[forestsynch].cluster-ok-recd(C, i) — true, then Vq £ nodes(C), 
s[clustersynch(C)].ok-recd(q,i) = true. 
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Both lemmas follow from operational proofs which are omitted here. 

Using these lemmas, we will show a possibilities mapping between CF and locsynch which will 
prove that every behavior of CF is a behavior of locsynch. Since the behaviors of CF are plainly 
those of CF projected onto locsynch, we will have proved the correctness of the implementation. 

Theorem 3.4 Consider f defined as follows: Vs £ states(CF), 
/(s) = {£ £ states(locsynch) \ Vp £ V 

t.gosent(p,i) — s[clustersynch(C)].gosent(p,i) and 
t.ok.recd(p, i) — s[cluster synch(C)].okjrecd(p, i), 

where C is the cluster such that p £ nodes(C)} 
Then f is a possibilities mapping from CF to locsynch. 

Proof: If s £ start(CF) then the existence of a start state in locsynch consistent with / is easily 
verified. Let s' be a reachable state of CF, t' any state in f(s') and (s 1 , n, s) a step of CF. We need 
to produce an extended step of locsynch (t 1 , 7, t) such that t £ f(s) and j\ext(LS) = n\ext(CF). 
We have the following cases depending on n: 

• iv = ok(p,i) for p £ V. Then take 7 = it. Since ok(p, i) is an input action, for some t, 
(t 1 , 7, t) is a step of LS. From the code describing the transitions of iSand cluster synch(C) 
(as above), and given that t' £ f(s') we can infer that t £ f(s). 

• 7r = go(p, i) for p £ V. Again 7 = n. If go(p, i) is an enabled output action in s', it is an 
enabled action in locsynch as well: We need to show that t' .gosent(p, i) — false and that Vg 
such that q — p or q £ neigh(p), t'.ok-recd(q, i) = true. The first condition is guaranteed by 
the mapping. We establish the truth of the second proposition now. Since go(p, i) in enabled 
in CF, s'[clustersynch(C)].cluster.gojrecd(i) = true. 

From Lemmas 3.2, 3.3 and 3.1, the corresponding ok-recd variables in t'[locsynch] are true. 
go(p, i) is therefore enabled when locsynch is in state t'. Let t be the state after go(p, i) has 
been performed. Clearly t £ /(s), and the mapping is legal. 



• 



7r = cluster _go(*,*) or cluster _ofc(*,*). Since these are both hidden actions, the mapping 
must leave t unchanged, with 7 being a sequence of zero actions. This is appropriate as neither 
action affects variables of the form s[*].gosent(*, *) or s[*].okjrecd(*, *), which determine the 
mapping. 

■ 
After having thus shown that behaviors of CF project onto locsynch as the latter automaton's 
behavior, we proceed now to argue that fair behaviors of CF will generate fair behaviors of locsynch 
too. 

3.3 Fairness of Implementation 

Let (5 be a behavior of CF interacting with clients, their front-ends and all the link automata, 
and 7 be such that 7 = fi\locsynch is a behavior of locsynch. As before, we let T be the unique 
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execution that 7 is the behavior of (T's uniqueness is a consequence of the implementing automata 
being deterministic, and of their not having internal actions). 

We will show that Ts being unfair will imply that (3 is not a fair behavior. We thus derive a 
contradiction to our assumption that /3 is a fair behavior of CF, and complete the fairness analysis. 

Theorem 3.5 If (3 is a fair behavior of CF interacting with clients, their front-ends and the 
bridging link automata, then 7 = j3\locsynch is a fair behavior of locsynch. 

Proof: We know from Theorem 3.4 that 7 is a behavior of locsynch. Assume that T the unique 
execution of that has 7 as its behavior, is not a fair execution of locsynch. Then there is a smallest 
i such that for some p 6 V, go(p, i) is enabled, but never occurs in T. 

Since for all j < i, go(q, j) occurs in Y for each q in V, the action client dnput(q, *,i - 1) is 
enabled at some point in /3, and will occur in (3. T. By Theorem 2.1 actions client -Output(q, *,i) 
actions therefore occur in f3 for each q £ V, and hence in T as well. 

Let p be in the cluster C in forest. If action go(p, i) does not occur in CF, it follows that for 
some node q in C or one of its neghboring clusters (that is, the set {r\ r £ nodes(C) such that 
C" G neighboring drees{C) U {C}}), ok(q, i) does not occur in V either, as otherwise f3 would be 
unfair too. But if ok(q,i) is never enabled in T even after client joutput{q,*,i) has occurred, it 
follows that client(qys packet meant for a neighbor is never acknowledgded. This however, can 
happen only if j3 is unfair, thus deriving a contradiction. B 

In subsequent sections we will provide distributed implementations of the cluster synch and 
forestsynch automata that are formal specifications of synchronizer (3 (at the intracluster level) 
and a (at the intercluster level) respectively, as described in Awerbuch's paper. Note that we 
can, in principle, use the distributed versions of arbitrary synchronizers to perform the task at the 
two levels. All we have to ensure is that the cluster. ok and cluster .go actions shared by the two 
specification automata can be communicated in the network between the implementations. In this 
paper we restrict ourselves to implementing a and f3 synchronizers that Awerbuch uses, where the 
problem of communicating is simply solved (as will be made clear shortly) by having the automata 
that share these actions run on the same physical node in the network. 

We close this section with a a brief overview of the remainder of the report. 

The next section will detail the distributed implementation of cluster synch automata, and the 
one after that, a mapping proof that will prove the implementation correct. This will be followed 
by an implementation of the automaton forestsynch, and the corresponding correctness proof in 
Sections 3.7 and 3.8. Finally, we end with a summary in Section 4. 

3.4 Intracluster Synchronization 

Our task in this section will be to give a distributed implementation of the intracluster synchronizer 
cluster synch(C) operating in a cluster consisting of nodes of a subtree C of the given spanning 
forest of G. 

Automaton cluster synch's function is to detect, after the commencement of each round, the 
condition that messages of the underlying synchronous algorithm sent out by each node in its 
cluster have been received. That is, detect the condition that the front ends at all the nodes in the 
cluster have output an ok action for that round. 
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Figure 4: (Non-)Root Node Automaton of the intracluster synchronizer 

At each non-root node of C, we have a component automaton intrajnonjroot of our distributed 
implementation sharing the ok with the front end at that node. Whenever an intra -non -root 
automaton learns that its front-end has sent its ok, and that the same is true of all the front ends 
at all descendent nodes, it communicates this fact to its parent in C. The process of conveying 
"ofcness" thus commences at the leaf nodes of C, and terminates at the root node of C. At the 
root node another component automaton (we shall refer to it as the "intra -root" automaton) waits 
until the convergecast terminates, and then outputs a cluster jok action indicating that all nodes 
in its tree have output their oks for that round. 

An additional task performed by cluster synch is to communicate, to all the nodes in its 
cluster, the receipt of the cluster-go action from the intercluster synchronizer. This is realized by 
broadcasting a message indicating receipt down the tree, from parent nodes to daughter nodes, 
until all the leaves have received this information. 

We now give a concise description of the the compostion's functions in terms of actions ex- 
changed between constituent components. 

The intrajroot automaton at the root node of each cluster waits until all the ok's of the nodes in 
the cluster have been convergecast to it, and then output a cluster-ok. Upon receiving a cluster-go, 
it broadcasts this information down all subtrees. The intra -non -root automata, one at every non- 
root node of a cluster, assist in the above by conveying the o&ness of the front end at that and at 
all descendent nodes up the tree, and by broadcasting the subtree -go message output by the root 
node when intrajroot receives a cluster-go (see fig 4.) 

If C is a subtree and p £ nodes(C), let children(p, C) denote p's children in C, and parent(p, C) 
be p's parent in C (if p is not the root of the subtree). Also, let root(C) denote the root node of C. 

We provide formal specifications of the described automata. 
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8.4.1 Root node automata of the intracluster tyuchroniaer 
Automaton Intrajrootfa C), p = rootfC) 

Signature 

Inpnt: 
oh{p,i) 

iuUr«ejoh ,n, (q,p, i), q € childrtnfr, C) 
dn*Ur^fo(C,i) 

Output: 

itbtn*4o' mt {p,q,i), q € cWWr«n(|», C) 
eiu*Urjak(C,i) 

Internal: 



State 

array ohjrecd(i) 

array go.j$ent(i) 

array $ubtrecjokjrecd{q,i)^ q e c&«Mren(|>, C) 

array *uWree-0O-*eni(?, »), 9 € eWWren(f», C) 

array cJ«*ter jo*_*ent(») 

array cluster 40 jrecd(i) 
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Transitions 



ok(p,i) 
Effect: 

s.okjrecd(i) = true 

subtree ^ok'" r (q,p, i) 
Effect: 

s.subtree-ok-recd(q, i) = true 

cluster -ok(C \i) 
Precondition: 

Vg £ children(p, C), s' .subtree-ok.recd(q,i) = true 

s .okjrecd(i) = true 

s' .cluster -oksent(i) = false 
Effect: 

s .cluster -oksent(i) = true 

cluster ~go(C \ i) 
Effect: 

s .cluster -go-recd(i) = true 

go(p,i) 

Precondition: 

s' .cluster -go-recd(i) — true 

s' .go.sent(i) = false 
Effect: 

s.gosent(i) = true 



subtree-go ™ 1 (p, q, i) 

Precondition: 

s' .cluster -go -recd(i) = true 
s' .subtree -go sent(q,i) = false 

Effect: 

s.subtree-gosent(q, i) = true 



Partitions 

All go(p, *) actions in one class, 

all subtree -go out {p, q, *) actions in one class, for each q £ children(p, C), 

all cluster jok(C , *) in one class. 

3.4.2 Non-root node automata in the intracluster synchronizer 
Automaton Intra.-nori-root(p, C), p £ nodes(C), p ^ root(C) 

Note that the only way intrajroot and intrajnonjroot differ is that the former has cluster-go 
as an input, while the latter has subtree-go instead, and that the root automaton has cluster .ok 
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as an output unlike intrajnoruroot, which has subtree-ok instead. Functionally however, the two 
actions within each pair are identical, differing only in their automata of origin or destination. 

Input: 
ok(p,i) 

subtree-ok' np (q,p,i), q G children(p,C) 
subtree-go'" p (r,p,i), r =parent(p,C) 

Output: 

9°(P, i) 

subtree-go° ut (p, q, i), q £ children(p, C) 

subtree-ok"" 1 (p, r,i), r = parent(p, C) 

Internal: 
none 

State 

array ok-recd(i) 

array gosent(i) 

array subtree -ok-recd(q, i), q 6 children(p, C) 

array subtree-go^sent(q,i), q £ children(p,C) 

array subtree joksent(i) 

array subtree -go-recd{i) 
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Transitions 



ok(p,i) 
Effect: 

s.okjrecd(i) = true 

subtree-ok' np (q, p, i) 
Effect: 

s.subtree-okjrecd(q, i) = true 

subtree-ok out (p, r, i) 
Precondition: 

Vg £ children(p,C), s' .8ubtreejokjrecd{q,i) = true 

s' .ok-recd(i) = true 

s .subtree-oksent(i) — false 
Effect: 

s .subtree-oksent(i) = true 

subtree.go'" p (r, p, i) 
Effect: 

a. subtree .go jrecd(i) = true 

go(p, i) 

Precondition: 

s' .subtree -go -recd(i) = true 

s .gosent(i) = false 
Effect: 

s.gosent(i) = true 

subtree -go out (p, q, i) 
Precondition: 

s' .subtree _go jrecd(i) = true 

s .subtree-.gosent(q,i) = false 
Effect: 

s.subtree-gosent(q,i) = true 



Partitions 

All go(p, *) actions in one class, 

all subtree -ok out (p, r, *) actions in one class, for r — parent(p), 

all subtree-go™ 1 (p,q, *) actions in one class, for each q £ children(p,C). 

We model the tree edges with I/O automata, and describe a simple automaton that represents 
an asynchronous bidirectional link between a parent and a daughter node that carries messages 
between their intra( -non) -root automata. 

3.4.3 Tree link automata 

Automaton Intra dree linkup — *■ q, C), p = parent(q, C) 
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Signature 

Input: 

subtree-go " (p, q, i) 
subtree _ofc° ut (q,p,i) 

Output: 

subtree-go' np (p, q, i) 
3ubtree-ok'" F (q, p, i) 

Internal: 



State 

buffer a first-in first-out queue of elements of arbitrary type, initially empty. 



Transition 



subtree-go out (p, q, i) 
Effect: 

Add go(q, i) to buffer 

subtree _ofe ou (q,p, i) 
Effect: 

Add ok(q, i) to buffer 

subtree -goi i "' p (p, q, i) 
Precondition: 

first(3'.6u//er) = go(q, i) 
Effect: 

s.buffer = Test(s'.buffer) 

subtree -ok lnp (q,p,i) 
Precondition: 

first(s'.6u//er) = ok(q,i) 
Effect: 

s.buffer = rest(s'.buffer) 



Partitions 

All subtree -go inp (p, q, *) actions in one class, 
all subtree-ok inp (q,p,i) actions in one class. 

In the next section we justify using the described collection to implement cluster synch by 
providing a proof of implementational correctness. 

3.5 Proof of implementation 

We will now show that the intra_root automaton, working with the intra_non_root automata and the 
links, implements cluster synch(C). Let TRA(C) be the composition of the intra-root(p,C), p — 
root{C) automaton with all the intrajnon-root(*, C) automata and the intra -tree J ink (* — ^ *,C) 
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automata. Further, let intra(r, C) denote intra.root(r, C) or intra -non. root(r, C) depending on 

whether or not r — p = root(C). 

Let TRA(C) = hide^TRA(C), where 

S = {it G acts(T 'RA(C))\tv is a subtree-ok inp , subtree.ok out , subtree.go inp , or subtree-go™ 1 

action} 

The following theorem guarantees that every behavior of TRA(C) is a behavior of cluster synch(C). 



Theorem 3.6 Vs G state s {T RA{C)) let f be as below. 

f(s)= {t G states(clustersynch(C))\ W G nodes(C) 

t.ok-recd(r,i) = s[intra(r,C)] .ok-recd(i) 
t.gosent(r,i) = s[intra(r,C)] .gosent(i) 
t.cluster-oksent(i) = s[intra-root(p,C)]. cluster -oksent(i) 
t.cluster-go-recd(i) = s[intrajroot(p,C)]. cluster .go -recd(i) } 

Then f is a possibilities mapping from TRA(C) to cluster synch(C). 

Proof: 



If s G start(T RA(C)) the existence of the start state in cluster synch(C) consistent with / is 
easily verified. Let s' be a reachable state of TRA{C) , t' any state in f(s') and (s', n, s) a step 
of TRA(C). We need to produce an extended step of cluster synch(C) {t',-y,t) such that t G f(s) 
and -y\ext(clustersynch(C)) = ir\ext(TRA(C)). 

• tv G S 

Taking 7 to be the null sequence of actions, and noting that for all steps (s', 7r, s) such that 
7r G S f(s') = f(s), we have / being a valid mapping, as t = t' G /(«') = f{s) 

• n G inputs(clustersynch(C)) 

We take 7 = 7r. Since 7 is an input, it follows that (t',j,t) is a step (t is the state 
cluster synch(C) winds up in after 7). That t G f{s) can be verified by inspection in both 
the two cases (n = ok(p, i) or cluster ~go(C, i)). 

• 7r = 50(7*, i), r G nodes(C) 

As before, take 7 = tt. li r — p, legality of / follows directly from the code implementing this 
action in locsynch and intra-root(r,C). If not, tv is output by intrajnon..root(r,C), which 
would imply that s' [intra -non .root(r, C)].subtree-go-recd(i) = true and 

s'[intra-non-root(r,C)].gosent(i) = false. From the second equality, we immediately have 
t'[cluster synch(C)].go^sent(p, i) = false. All we have to show is that when cluster synch(C) 
reaches state t' it has already received the cluster-go for the zth round. 

Claim 3.1 For any r G nodes(C), r / root{C), let q^ j = 0, 1, ■ • ■ , d be the nodes in C such 

that q = r, q d — root(C) and Vj < d, q i+1 = parent(q j} C). 

IfTRA(C) is in state s' such that s'[intra-non-root(q ,C)].subtree-go-recd(i) = true, then 

1. Vj < d^s'[intra-non-root{q^C)\.subtree-go-recd{i) = true 

2. s'[intra-root(q d ,i)].cluster-go-recd(i) = true. 
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Proof of claim: 

1. By induction on j. 

2. From Claim 3.1.1 s' .[intra jnon-root(q d _ t , C)].subtree-gojrecd(i) = true. From the im- 
plementation of the intrajroot automaton, the claim follows. 



Action 7 will thus be enabled in t' and t such that (t', 7, t) is a step of cluster synch(C) will 
be in f(s). 

n = cluster _ok(C , i) 

Taking 7 = n. If intrajroot receives subtree-ok messages from all its children, it needs to 
be shown that at the intrajnon_root automata at all descendent nodes, the okjrecd received 
variable for that round is true. That 7 will be enabled in cluster synch(C) will follow imme- 
diately, and the proof would be done. 



Claim 3.2 W G states(T RA(C)) if s '[intra _root(p,C)].clusterjoksent(i) = true thenar G 
nodes(C), s'[intra(r,C)].subtree-oksent(i)] = true 

Proof of claim: For any node r in C, let depth(r) denote its distance from root(C), 
along the unique path entirely in C between r and root(C). We prove the claim by induc- 
tion on the depth of a node r. For a node r at depth 1, since r is a child of the root node, 
s'[intrajroot(p, C)].cluster -oksent(i) = true implies that s'[intra(r, C)}.subtreejoksent{i) = 
true. 

Assume that at any node r' at a depth d - 1, s'[intra(r', C)].subtreejoksent{i)] = true. 
Consider a node r at a depth d, which has q at depth d - 1 as the parent. The required 
condition follows directly from the implementation of the subtree jok mit {r, q, i) action. 
■ 

From the above claim and the preconditions of the subtreejok™ 1 action we have, for all nodes 
q, s'[intra(q, C)].okjrecd(i) = true, as required. 



We are thus guaranteed that behs(TRA(C)) C behs(cluster synch(C)) . But behaviors of 
TRA(C) are clearly projections of behaviors of TRA(C) onto cluster. synch(C). Hence, TRA{C) 
implements cluster synch(C). 

3.6 Fairness of Implementation 

We now argue that TRA is a fairness preserving implementation. Automaton clustersynch(C), for 
some subtree C in forest, has cluster jok{C , *) and go{p, *) for each p G nodes(C) as fairness par- 
titions. As in earlier fairness arguments, it can be shown that unfair behaviors of cluster synch(C) 
can be generated only from unfair behaviors of TRA(C). 

Theorem 3.7 Let (3 be a fair behavior of TRA(C), for some subtree C in forest. Then 7 = 
/3| cluster synch(C) is a fair behavior of automaton cluster synch(C). 
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Proof: 

Theorem 3.6 guarantees that 7 is a behavior of cluster synch(C). Assume that 7 is an unfair 
behavior of cluster synch(C), and let T be the unique execution of that automaton which has 7 as 
its behavior. 

Consider the case when the action go(p, i) is enabled in T but never performed, for some i and p 
in V. If p is the root node of the subtree C, a contradiction is immediate: The enabling of go(p, i) 
implies that cluster -go(C, i) occurs in 7, and hence in /3 as well. In that event, go(p, i) is enabled 
in the execution corresponding to /3 as well, in the automaton intrajroot(p,C). Since (3 is a fair 
behavior, this results in that action being performed, which is a contradiction. 

If p is a non-root node of C, at a depth d > 1 from the root node, a inductive argument yields 
a contradiction: If go(p,i) does not occur in the fair behavior /3 of TRA(C), it follows that it is 
never enabled during that execution. Automaton intrajnonjroot{p, C) therefore never receives a 
subtree-go message from the automaton intra(parent(p,C),C) at level d — 1. This can happen 
only if intra(parent(p,C),C) never receives a subtreejgo from its parent. Thus if p is such that 
go(p, i) is enabled but never performed in T, then it follows by induction that the the intra -non -root 
automaton at depth 1 on the path joining the root to p never receives a subtree-go message. Since 
this can happen only if cluster -go(p, i) is not performed, we have a contradiction. 

We now consider the case when, for some i, the action cluster -ok(C,i) is enabled, but never 
performed in T. If cluster -ok is enabled in cluster synch(C), then it follows that all the front-ends 
in C have sent in their oks for that round to cluster synch(C). Thus all these oks appear in (3 as 
well. 

If cluster -ok is not performed in the execution corresponding to f3 (in TRA(C)), it would have 
to have been disabled throughout that execution, for otherwise /3 would not be fair. This implies 
that one of the root node's child nodes never sends in its subtree-ok action for that round. By the 
same argument, if a node at depth d does not ever receive a subtree-ok from one of its children for 
some round, this can happen only if that child node did not receive such an action from one of its 
children, at depth d + 1. 

This inductive argument leads to the fact that at least one leaf node intra -non -root automaton 
never sends in a subtreejok action for that round. But every descending branch of the subtree ends 
in a leaf node whose subtreejok action is enabled if all the oks for that round appear in /3, and, if 
/3 is fair, the automaton at this node does perform the subtree-ok action at some point in (3. We 
thus derive a contradiction to our assumption that fair behaviors of TRA(C) can generate unfair 
behaviors of cluster synch(C). ^ 

We saw in an earlier section that locsynch can be implemented as a combination of an intra- 
cluster synchronizer (as described in this section) and an intercluster synchronizer that synchronizes 
between clusters. We now direct our attention to the latter automaton, and put together a collection 
of automata that will implement the intercluster synchronizer forestsynch fairly. 

3.7 Intercluster Synchronization 

We will describe a collection of automata that will implement forestsynch. This automaton's 
function resembles that of locsynch rather closely: It detects the condition that a cluster and all 
its neighboring clusters are "safe" (have sent in their cluster-oks) and communicates this fact to 
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Figure 5: The root-node component automaton of the intercluster synchronizer. 

the intracluster synchronizer for that cluster. 

This requires each intracluster to share actions with a component of for est synch's distributed 
implementation. In our (and Awerbuch's) implementation, this constraint is met by having a key 
component automaton (that we shall call "inter -root") of forest synch's implementation run on 
each root-node of a cluster (which has the intra -root automaton that generates the cluster -ok 
action, and receives the cluster -go action). 

Communication between any two neighboring clusters in the distributed implementation occurs 
through a predetermined set of edges that form a path between the root nodes of the two cluster. 
All but one of the path's edges are chosen from the spanning trees of either cluster. We call 
the non-tree edge, the ■prefer red link between the two clusters, and denote the set of all such 
cluster-bridging edges by pr e f erred Jinks( for est). 

We now describe the operation of the distributed implementation. 

The distributed implementation of forestsynch consists of the following in each cluster: a 
inter-root automaton at the root of the cluster, and a inter -non -root automaton at each non-root 
node of the cluster. 
The functions of the inter-root automaton are twofold. 

• Upon being told by cluster synch that the cluster is safe, it will broadcast this down to the 
leaves, through the subtrees, so that neighboring clusters can get this information via bridging 
preferred links. 

• Messages from neighboring clusters telling the cluster in question that they are safe are 
convergecast to the inter-root automaton. After checking to see if its own cluster is safe, it 
will then give the cluster synch automaton (also at the root node) a cluster-go. 

Figure 5 summarizes the inter -root automaton's functioning pictorially. 

The inter -non-root automaton's functioning resembles that of inter-root closely. Upon receiv- 
ing relay jokjrecd passed down to it along the path joining it to inter-root, inter -non -root passes 
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Figure 6: The non-root node component automaton of the intercluster synchronizer. 

it along to its children, and any preferred links that may be incident upon it. This behaviour corre- 
sponds to that exhibited by the root automaton when it receives a cluster safe from cluster synch. 
Another important task performed by inter jnonjroot is to help convergecast readiness to the root. 
An inter jnoruroot automaton is ready if 

• All its children are ready and 

• if it has received relay -duster -oks along all preferred edges incident upon it. 

This is merely the condition that the cluster itself, all clusters neighboring it and (neighboring) 
all its descendant nodes are known to be safe. A pictorial representation of the inter jnon -root 
automaton is given in fig 6. 

Working together, these automata implement forestsynch. As before, we give the formal 
specifications of the automata described above, prove the implementation correct, and conclude 
with a few remarks arguing that all fair executions of the implementation generate fair executions 
of the implemented system. 

3.7.1 Root node automata in the intercluster synchronizer 

Automaton inter jroot(p,C), p = root(C), where C is a subtree of forest. 

We describe the component automaton of forestsynch that runs on the root node of every subtree 

C of forest. 

Input: 

cluster -ok(C ', i) 

relay -cluster -.ok tnp {q,p, i), (p, q) £ preferred-links(forest) 
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ready lnp (q, i), q £ children(p) 
Output: 

cluster -go(C, i) 

relay .cluster jok out (p,q,i), q £ children(p) or (p,q) £ preferredJinks(forest) 
Internal: 



State 

array cluster -ok-recd(i) 

array relay -ok-recd{q, i), (p,q) £ preferredJinks(forest) 

array ready-recd(q,i), q £ children(p) 

array cluster -gosent(i) 

array relay-oksent(q, i), (p,q) £ pre f erred Jinks( for est) 



Transition 



cZu3ter_o/:(C, i) 
Effect: 

s.cZuster-ofc_rec<i(z) = true 

reZaj/-cZuster_ofc ,np (g,p, i) 
Effect: 

s.reZay J ofc_recd(g, i) = true 

ready' np (q, i) 
Effect: 

s. ready jrecd(q, i) = true 

cluster -go(C,i) 
Precondition: 

Vg such that (p, q) £ preferredJinks(forest), s' .relay jok-recd(q,i) = true 
Vg £ c/iiZifrera(p), a'.rea<ii/_reccf(g, i) = true 
a .cZuster_ofc_rec(f(i) = true 
s'.cZuater_<7o_sent(t) = false 
Effect: 

s.cZuater_go_sent(i) = true 

reZaj/_cZuater_ofc°"'(p, g, i) 
Precondition: 

s .cluster -okjrecd(i) = true 

s .relay -oksent(q,i) — false 
Effect: 

s.relayjoksent^q, i) = true 



Partitions 



All cluster-go(C, *) actions in one class, 

all relay -duster -ok out (p, q, *) actions in one class, for all p and q such that (p, q) £ pre f erred Jinks(forest). 
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We now describe, in formal terms, the component automaton on each non-root node in a subtree 
C of forest. 

3.7.2 Non-root node automata in the intercluster synchronizer 

Automaton inter -non -root(p, C), p G nodes(C), p / root(C) and C is a subtree of forest 

Signature 

Input: 

relay jcluster-ok lnp (r,p, i), r — parent(p) or (p,r) EpreferredJinks(forest) 

ready'" p (q,i), q £ children(p) 
Output: 

ready° ut (r, i), r = parent(p) 

relay ^cluster -ok"" i (p,q,i), q £ children(p) or (p,q) EpreferredJinks(forest) 
Internal: 



State 

array cluster jokjrecd(i) 

array relay-okjrecd(q,i), (p,q) G preferred Jinks(forest) 

array ready -recd(q,i), q G children(p) 

array ready -.sent(i) 

array relay-oksent(q, i), (p,q) G preferredJinks(forest 
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Transition 



relay -duster -ok'" p (r,p, i) 
Effect: 

if r = parent(p) then s. cluster jokjrecd(i) = true 
else s.relay-okjrecd(r,i) = true 

ready' np (q, i) 
Effect: 

s.ready_rec<i(g,i) = true 

ready e "* t (r,i), r = parent(p) 
Precondition: 

Vg such that (p,q) S preferredJinks(forest), s' .relay jokjrecd(q,i) = true 

Vg 6 children(p), s' .ready -recd(q,i) = true 

s' .cluster jokjrecd(i) = true 

s .ready sent(i) = false 
Effect: 

s.readysent(i) = true 

relay. cluster. ok° ut (p, q, i) 
Precondition: 

s .cluster Jok-recd{i) = true 
s' .relay-oksent(q,i) = false 
Effect: 

s.relayjoksent(q, i) = true 



Partitions 

All ready™ 1 (r, *) in one class, r = parent(p), 

all relay .cluster .ok out (p, q, *) actions in one class, for all p and q such that (p, q) 6 pre f erred Jinks(forest). 

As before, we model asynchronous communication channels by I/O automata, and provide, 
in the next two subsections, formal specifications of link automata which represent tree edges 
(inter Jtree dink automata) and those that represent preferred links (preferred Jink automata) 

3.7.3 Tree link automata 

Automaton Inter Jtree dink(p — ^ q, C), p = parent(q, C). 

This automaton allows inter -non -root (p, C) or inter jroot(p, C) and inter -non -root(q, C) to ex- 
change messages. 

Signature 

Input: 

ready°" t (q, i) 

relay -cluster -ok out (p,q,i) 
Output: 

ready inp (q,i) 

relay .cluster -ok' np (p, g, i) 
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Internal: 
none 

State 

buffer a fifo queue of any type 

Transition 



ready™* (q, i) 
Effect: 

Add ready(i) to buffer 

relay -duster -ok oai (p,q, i) 
Effect: 

Add ok(i) to buffer 

ready lnp (q, i) 
Precondition: 

first(*'.fcu//er) = ready(i) 
Effect: 

s.buffer = icst(s'.buffer) 

relay -cluster joh inp (p, q, i) 
Precondition: 

first(s'.&u//er) = ok(i) 
Effect: 

s.buffer = rest(a' .buffer) 



Partitions 

All ready inp (q, *) actions in one class, 

all relay -duster jok mp (p, q, *) actions in one class. 

3.7.4 Preferred link intercluster automata 

Automaton Preferred Jink(p — ^ r, C p , C r ), p 6 C p , r £ C r , C p and C r distinct trees in forest. 
As mentioned elsewhere, preferred links assist in intercluster communication: they form the com- 
munication channels through which each cluster conveys its "ofcness" to neighboring clusters. 

Input: 

relay -cluster -ok * 1 (p, r, i) 

relay .cluster -ok "* (r, p, i) 
Output: 

relay .cluster -ok xrxp (p, r, i) 

relay .cluster -ok xnp (r ,p, i) 
Internal: 
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State 

buffer a first-in first-out queue of elements of arbitrary type, initially empty. 



Transition 



relay -duster -ok° ut (p, r, i) 
Effect: 

Add ok(p,r,i) to buffer 

relay -duster -ok ou (r,p, i) 
Effect: 

Add ok(r,p, i) to buffer 

relay -duster -ok' np (p, r, i) 
Precondition: 

first(s'.6u//er) = ok(p, r, i) 
Effect: 

s.buffer = rest(s' .buf fer) 

relay -duster jok ,np (r, p, i) 
Precondition: 

first(s .buffer) = ok[r,p, i) 
Effect: 

s.buffer = Test(s'.buffer) 



Partitions 

All relay -duster -ok tnp (p,r, *) actions in one class, 
all relay -.cluster -X>k mp (r,p, *) actions in one class. 

Having thus described the distributed implementation of forestsynch, we have now to prove 
correctness, as before. We again use a possibilities mapping proof to show behavioral inclusion. 

3.8 Proof of implementation 

Let TER denote the composition of all the inter -root automata, all the inter jnoruroot automata 
and all the link automata. Take S = {tv £ acts(TER) \ n is an action that is not of the form 
cluster. go(*, *) or cluster -.ok (*, *)} and TER = hide^TER. Further, let TER(C,p), p G nodes{C) 
denote the inter -{non)jroot automaton at the node p in C. 



Theorem 3.8 Vs G states(TER), 

f(s) = {t £ states (forest.synch( forest)) \ VC, a subtree in forest, 



t.cluster-ok-recd(C,i) — s[T E R(C,root(C) )} .cluster -ok-recd(i) 
t.cluster-gosent(C,i) = s[TER(C,root(C))].cluster-gosent(i) 



Then f is a possibilites mapping from TER to forestsynch. 
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Proof: If s £ start(TER), the existence of the start state in forestsynch consistent with / 
is easily verified. Let s' be a reachable state of TER, t' any state in f(s') and (s', n, s) a step 
of TER. We need to produce an extended step (t', 7, i) of forestsynch such that t £ f(s) and 
j\ext(forestsynch) = ir\ext(TER). We distinguish two cases depending on n, and take 7 to be n 
in both cases. 

If 7r = cluster jok(C ', z) for any C, a cluster, the proof of correctness of / is immediate as n is 
an input action with the right effects in both automata. Otherwise, n = cluster -go(C,i). In this 
case the only non-trivial argument we have to make is to show that if cluster-go is enabled in TER 
then the intra- root automaton in all the neighboring clusters have performed the cluster -ok action 
for that round. 

Consider any neighboring cluster C" such that the preferred link between C and C" is not 
incident upon either of their root nodes. If C" is not safe, we show that one of root (C)'s children 
will not be ready - to be precise, the child node on the path containing the preferred link between 
the two clusters will have not sent a ready action yet. 

We know there exists a sequence of nodes u , U\, ..., u n , v n >, v n i_i, .., i>i, ^o such that the Uj's 
form a path from no = root(C) to a leaf of C, the Uj's form a similar path in C from v = root(C ) 
and the edge (u n , v n >) between the two leaves is a preferred link in G. 

If, in state s', s'[TER(C',v )].clusterjok-recd(i) ^ true, then by the implementation of the 
inter-non-root automata at the descendent nodes, s'[TER(C, vi)].clusterjok-recd(i) ^ true. With 
this as the base case, an inductive argument shows that for all Vj, < j < n', 
s'[TER(C',Vj)].clusterjok-recd(i) ^ true. We exploit this fact to complete the proof. 

If s'[TER(C',v n >)].cluster-okjrecd(i) = false, then the automaton on the other side of the 
preferred link (across from TER{C',v n i)) could not have received a relay -.cluster jok from the 
inter-(non-)root automaton at v n , that is, s'[TER(C, v n )]. relay jok-recd{v n i, i) / true. 
Thus s'.[TER(C, v n )].readysent(i) is false, as is s'.[TER{C, v n -i)].readyjrecd(v n , i). By induc- 
tion on smaller values of j, for all j > 0, s'.[TER(C',Vj)].readyjrecd(vj + i,i) — false. Hence we 
see that if s'[TER(C, root(C))].readyjrecd(q, i) is true for all q £ children(root(C)), then all the 
neighboring clusters considered above will have sent their cluster joks for that round already. 

In the case when the preferred link between C and C does touch either of their roots, the above 
proof lends to an easy modification, and we can conclude that if all the relay jok.recd actions and 
ready -reed actions are in through the preferred links, then none of the neighboring clusters can be 
unsafe. 

The correctness of / is thus guaranteed, and hence behs(TER) C behs{f orestsynch) . 

m 

From the preceding theorem, we can conclude that TER implements forestsynch correctly, 
as both TER and TER project identically on the automaton forestsynch. We turn our attention 
to fairness issues next. 

3.9 Fairness of Implementation 

The composition automaton TER has the following property: Fair executions of TER (or equiv- 
alently, executions that are fair to each of its component automata) generate fair behaviors of 
forestsynch. We outline below the argument supporting this claim. 
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Automaton forest-synch's only locally controlled class of actions consists of actions of the form 
cluster -go{C , *), where C is a subtree of forest. 

As before, we need to show that the generated executions of forestsynch are fair to each of these 
classes of actions. It will be shown that unfair behaviors can be generated only from executions 
of the implementation that are unfair to the automata that generate the cluster-gos. We are thus 
guaranteed that fair behaviors of TER will generate just fair behaviors of forestsynch. 

Theorem 3.9 Let ft be a fair behavior of TER. Then 7 = ft\ forestsynch is a fair behavior of 
that automaton. 

Proof: Theorem 3.8 guarantees that 7 is a behavior of forestsynch. Assume that 7 is an unfair 
behavior of forestsynch, and let T be the unique execution of that automaton which has 7 as 
its behavior. Then there is a subtree C, and a smallest i such that the action cluster-go(C, i) is 
enabled in Y but never performed in either T or ft. 

If cluster -go(C, i) is enabled in T, then, for all subtrees C" such that C is a neighboring cluster 
of C or C itself, the action cluster jok(C',i) must occur in T. Using the fact that ft is fair, an 
inductive argument will show that the occurrence (in ft) of each cluster -ok(C',i) at a neighboring 
cluster will cause a relay -duster -ok(p,q,i) action to be performed, for some p in nodes(C ) and 
q in nodes(C) (the induction is on the distance of each node from root(C) on the unique path 
between root(C') and q, the node in C across a preferred-link from a node in C"). 

We are thus guaranteed that ii ft is fair, but does not contain cluster ~go(C, i), it is because some 
inter -non -root automaton at one of 7*ooi(C)'s child nodes has not performed a ready° ut action. A 
second inductive argument can now be used to show that there exists a sequence of nodes in C, 
v , vi, ■ ■ -vj, such that v = root(C), u,_i = parent(vi), for i = 1, • • -d, and vj, is a leaf node of C, 
with the property that no ready out (vj, i) occurs in ft, for 1 < j < d. 

But readj/ out (U(i, i) is enabled in /3: the relay -duster joks occur in ft if there are incident 
preferred-links onto vj, and we already know that all the relevant cluster -oks occur in ft, thus 
enabling that action. If ft is fair, this action will then occur in that sequence, thus yielding a 
contradiction. 



We thus have a node-level implementation of a collection of automata, whose composition 
synchronizes clients in a provably correct way. 

We end this report in the next section with a summary of our efforts and some noteworthy 
features of this proof of correctness. 

4 Summary 

In this paper we have offered a formal, rigorous proof of the correctness of Awerbuch's algorithm 
for network synchronization. We specified both the algorithm and the correctness condition using 
the I/O automaton model. Since the algorithm uses simpler algorithms for synchronization within 
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and between 'clusters' of nodes, our proof could have imported as lemmas the correctness of these 
simpler algorithms, had their correctness been proved elsewhere. Alternatively, the understanding 
of the modularity that the proof gives us would allow us to see how to safely change the choices of 
implementation of the separate parts of the synchronizer, independently of one another. Also, we 
clearly benefit from having carried out the correctness proof in the I/O automaton model which 
supports modularity, since the network synchronizer is often used as an 'off-the-shelf building block' 
component in a larger system, and proofs of the correctness of the system will be able to use our 
proof without change. 
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